Motion blur compensation through eye tracking

ABSTRACT

A user&#39;s eyes and if desired head is tracked as the user&#39;s gaze follows a moving object on a display. Motion blur of the moving object is keyed to the eye/head tracking. Motion blur of other objects in the frame also may be keyed to the eye/head tracking.

FIELD

The present application relates generally to motion blur compensationthrough eye tracking.

BACKGROUND

Motion blur is typically computed by the motion of an object across ascene. Specifically, for a fast-moving object moving from left to right,a game's renderer may choose to “streak” the motion across the screen.

To achieve motion blur, only camera motion may be accounted for tocreate a radial blur full-screen, or more selective per-object motionblur may be used by programming a software shader to create a velocitybuffer to mark motion intensity for a motion blurring effect on theobject. The motion blurring effect causes the object to appear“streaky”.

SUMMARY

As understood herein, current motion blur techniques do not account foreye motion (and sometimes the head motion) of the player/viewer. Thismeans that a moving object may be rendered with motion blurring eventhough the viewer may be following the object with his gaze, in whichcase in real life the object would not seem motion-blurred.

Accordingly, a system includes at least one computer medium that is nota transitory signal and that in turn instructions executable by at leastone processor to execute gaze tracking of a user viewing a display andimplement at least one motion blur feature of at least one objectpresented on the display according to the gaze tracking.

In some examples, the gaze tracking includes one or more of headtracking, eye tracking, and user body motion tracking. Stateddifferently, a holistic motion accounting may be implemented thataccounts for motion of the viewer's eyes, head, and virtual body (whichmay be sitting in a car/on a train.)

In some embodiments, the instructions can be executable to identify theobject as moving across the display, identify the gaze tracking asfollowing the object, and responsive to identifying the object as movingacross the display and identifying the gaze tracking as following theobject, implement the motion blur feature as being no motion blur of theobject.

In example embodiments, the instructions are executable to identify theobject as moving across the display, identify the gaze tracking as notfollowing the object, and responsive to identifying the object as movingacross the display and identifying the gaze tracking as not followingthe object, implement the motion blur feature as being motion blur ofthe object. In such examples the motion blur of the object may beproportional to a difference between a motion vector of the object and amotion vector of the gaze tracking.

In some implementations, the instructions can be executable to identifythe object as not moving across the display, identify the gaze trackingas moving relative to the display, and responsive to identifying theobject as not moving across the display and identifying the gazetracking as moving relative to the display, implement the motion blurfeature as being motion blur of the object.

A hardware implementation may be offered in which the motion blurfeature is implemented at least in part by establishing a firstbrightness and a first exposure in a first region of the displayincluding the object and establishing a second brightness and/or asecond exposure in a second region of the display not including theobject.

In another aspect, a method includes implementing first motion blurringof an object presented on a display in accordance with a user's gazefollowing the object as the object moves and implementing second motionblurring of the first object in accordance with the user's gaze notfollowing the object as the object moves.

In another aspect, an apparatus includes at least one display and atleast one processor configured with instructions to identify a firstmotion vector associated with a viewer of the display, identify a secondmotion vector associated with an object presented on the display, andimplement motion blurring of the object based on the first and secondmotion vectors.

The details of the present application, both as to its structure andoperation, can be best understood in reference to the accompanyingdrawings, in which like reference numerals refer to like parts, and inwhich:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example system including an example inaccordance with present principles;

FIG. 2 is a screen shot schematically illustrating a viewer whose pointof gaze (POG) remains on a fixed object presented on a display;

FIG. 3 is a screen shot schematically illustrating a viewer whose POGfollows a moving object presented on a display;

FIG. 4 illustrates example logic in example flow chart format consistentwith present principles;

FIG. 5 illustrates an example head-mounted display (HMD) schematicallyillustrating components of the HMD;

FIG. 6 illustrates an example hardware system implementation; and

FIG. 7 illustrates example logic in example flow chart format consistentwith FIG. 6 .

DETAILED DESCRIPTION

This disclosure relates generally to computer ecosystems includingaspects of consumer electronics (CE) device networks such as but notlimited to computer game networks. A system herein may include serverand client components which may be connected over a network such thatdata may be exchanged between the client and server components. Theclient components may include one or more computing devices includinggame consoles such as Sony PlayStation® or a game console made byMicrosoft or Nintendo or other manufacturer, virtual reality (VR)headsets, augmented reality (AR) headsets, portable televisions (e.g.,smart TVs, Internet-enabled TVs), portable computers such as laptops andtablet computers, and other mobile devices including smart phones andadditional examples discussed below. These client devices may operatewith a variety of operating environments. For example, some of theclient computers may employ, as examples, Linux operating systems,operating systems from Microsoft, or a Unix operating system, oroperating systems produced by Apple, Inc., or Google. These operatingenvironments may be used to execute one or more browsing programs, suchas a browser made by Microsoft or Google or Mozilla or other browserprogram that can access websites hosted by the Internet serversdiscussed below. Also, an operating environment according to presentprinciples may be used to execute one or more computer game programs.

Servers and/or gateways may include one or more processors executinginstructions that configure the servers to receive and transmit dataover a network such as the Internet. Or a client and server can beconnected over a local intranet or a virtual private network. A serveror controller may be instantiated by a game console such as a SonyPlayStation®, a personal computer, etc.

Information may be exchanged over a network between the clients andservers. To this end and for security, servers and/or clients caninclude firewalls, load balancers, temporary storages, and proxies, andother network infrastructure for reliability and security. One or moreservers may form an apparatus that implement methods of providing asecure community such as an online social website to network members.

A processor may be a single- or multi-chip processor that can executelogic by means of various lines such as address lines, data lines, andcontrol lines and registers and shift registers.

Components included in one embodiment can be used in other embodimentsin any appropriate combination. For example, any of the variouscomponents described herein and/or depicted in the Figures may becombined, interchanged, or excluded from other embodiments.

“A system having at least one of A, B, and C” (likewise “a system havingat least one of A, B, or C” and “a system having at least one of A, B,C”) includes systems that have A alone, B alone, C alone, A and Btogether, A and C together, B and C together, and/or A, B, and Ctogether, etc.

Now specifically referring to FIG. 1 , an example system 10 is shown,which may include one or more of the example devices mentioned above anddescribed further below in accordance with present principles. The firstof the example devices included in the system 10 is a consumerelectronics (CE) device such as an audio video device (AVD) 12 such asbut not limited to an Internet-enabled TV with a TV tuner (equivalently,set top box controlling a TV). The AVD 12 alternatively may also be acomputerized Internet enabled (“smart”) telephone, a tablet computer, anotebook computer, a HMD, a wearable computerized device, a computerizedInternet-enabled music player, computerized Internet-enabled headphones,a computerized Internet-enabled implantable device such as animplantable skin device, etc. Regardless, it is to be understood thatthe AVD 12 is configured to undertake present principles (e.g.,communicate with other CE devices to undertake present principles,execute the logic described herein, and perform any other functionsand/or operations described herein).

Accordingly, to undertake such principles the AVD 12 can be establishedby some, or all of the components shown in FIG. 1 . For example, the AVD12 can include one or more displays 14 that may be implemented by a highdefinition or ultra-high definition “4K” or higher flat screen and thatmay be touch-enabled for receiving user input signals via touches on thedisplay. The AVD 12 may include one or more speakers 16 for outputtingaudio in accordance with present principles, and at least one additionalinput device 18 such as an audio receiver/microphone for enteringaudible commands to the AVD 12 to control the AVD 12. The example AVD 12may also include one or more network interfaces 20 for communicationover at least one network 22 such as the Internet, an WAN, an LAN, etc.under control of one or more processors 24. Thus, the interface 20 maybe, without limitation, a Wi-Fi transceiver, which is an example of awireless computer network interface, such as but not limited to a meshnetwork transceiver. It is to be understood that the processor 24controls the AVD 12 to undertake present principles, including the otherelements of the AVD 12 described herein such as controlling the display14 to present images thereon and receiving input therefrom. Furthermore,note the network interface 20 may be a wired or wireless modem orrouter, or other appropriate interface such as a wireless telephonytransceiver, or Wi-Fi transceiver as mentioned above, etc.

In addition to the foregoing, the AVD 12 may also include one or moreinput and/or output ports 26 such as a high-definition multimediainterface (HDMI) port or a USB port to physically connect to another CEdevice and/or a headphone port to connect headphones to the AVD 12 forpresentation of audio from the AVD 12 to a user through the headphones.For example, the input port 26 may be connected via wire or wirelesslyto a cable or satellite source 26 a of audio video content. Thus, thesource 26 a may be a separate or integrated set top box, or a satellitereceiver. Or the source 26 a may be a game console or disk playercontaining content. The source 26 a when implemented as a game consolemay include some or all of the components described below in relation tothe CE device 48.

The AVD 12 may further include one or more computer memories 28 such asdisk-based or solid-state storage that are not transitory signals, insome cases embodied in the chassis of the AVD as standalone devices oras a personal video recording device (PVR) or video disk player eitherinternal or external to the chassis of the AVD for playing back AVprograms or as removable memory media or the below-described server.Also, in some embodiments, the AVD 12 can include a position or locationreceiver such as but not limited to a cellphone receiver, GPS receiverand/or altimeter 30 that is configured to receive geographic positioninformation from a satellite or cellphone base station and provide theinformation to the processor 24 and/or determine an altitude at whichthe AVD 12 is disposed in conjunction with the processor 24. Thecomponent 30 may also be implemented by an inertial measurement unit(IMU) that typically includes a combination of accelerometers,gyroscopes, and magnetometers to determine the location and orientationof the AVD 12 in three dimension or by an event-based sensors.

Continuing the description of the AVD 12, in some embodiments the AVD 12may include one or more cameras 32 that may be a thermal imaging camera,a digital camera such as a webcam, an event-based sensor, and/or acamera integrated into the AVD 12 and controllable by the processor 24to gather pictures/images and/or video in accordance with presentprinciples. Also included on the AVD 12 may be a Bluetooth transceiver34 and other Near Field Communication (NFC) element 36 for communicationwith other devices using Bluetooth and/or NFC technology, respectively.An example NFC element can be a radio frequency identification (RFID)element.

Further still, the AVD 12 may include one or more auxiliary sensors 38(e.g., a motion sensor such as an accelerometer, gyroscope, cyclometer,or a magnetic sensor, an infrared (IR) sensor, an optical sensor, aspeed and/or cadence sensor, an event-based sensor, a gesture sensor(e.g., for sensing gesture command), providing input to the processor24. The AVD 12 may include an over-the-air TV broadcast port 40 forreceiving OTA TV broadcasts providing input to the processor 24. Inaddition to the foregoing, it is noted that the AVD 12 may also includean infrared (IR) transmitter and/or IR receiver and/or IR transceiver 42such as an IR data association (IRDA) device. A battery (not shown) maybe provided for powering the AVD 12, as may be a kinetic energyharvester that may turn kinetic energy into power to charge the batteryand/or power the AVD 12. A graphics processing unit (GPU) 44 and fieldprogrammable gated array 46 also may be included. One or more hapticsgenerators 47 may be provided for generating tactile signals that can besensed by a person holding or in contact with the device.

Still referring to FIG. 1 , in addition to the AVD 12, the system 10 mayinclude one or more other CE device types. In one example, a first CEdevice 48 may be a computer game console that can be used to sendcomputer game audio and video to the AVD 12 via commands sent directlyto the AVD 12 and/or through the below-described server while a secondCE device 50 may include similar components as the first CE device 48.In the example shown, the second CE device 50 may be configured as acomputer game controller manipulated by a player or a head-mounteddisplay (HMD) worn by a player. In the example shown, only two CEdevices are shown, it being understood that fewer or greater devices maybe used. A device herein may implement some or all of the componentsshown for the AVD 12. Any of the components shown in the followingfigures may incorporate some or all of the components shown in the caseof the AVD 12.

Now in reference to the afore-mentioned at least one server 52, itincludes at least one server processor 54, at least one tangiblecomputer readable storage medium 56 such as disk-based or solid-statestorage, and at least one network interface 58 that, under control ofthe server processor 54, allows for communication with the other devicesof FIG. 1 over the network 22, and indeed may facilitate communicationbetween servers and client devices in accordance with presentprinciples. Note that the network interface 58 may be, e.g., a wired orwireless modem or router, Wi-Fi transceiver, or other appropriateinterface such as, e.g., a wireless telephony transceiver.

Accordingly, in some embodiments the server 52 may be an Internet serveror an entire server “farm” and may include and perform “cloud” functionssuch that the devices of the system 10 may access a “cloud” environmentvia the server 52 in example embodiments for, e.g., network gamingapplications. Or the server 52 may be implemented by one or more gameconsoles or other computers in the same room as the other devices shownin FIG. 1 or nearby.

The components shown in the following figures may include some or allcomponents shown in FIG. 1 .

FIG. 2 illustrates a display 200 presenting an animated moving object202, in the example shown, a plane, moving to the left as indicated bythe motion vector 204. A viewer 206 has a point of gaze (POG 208 lookingat a stationary object 210 on the display 200, in this case, a tree.Because in the example of FIG. 2 the POG is steadily fixed on thestationary object 210 and is not following the moving object 202, thePOG 208 has a motion vector of zero in the direction of travel of themoving object 202, i.e., in the x-dimension. Accordingly, motion blur,indicated by the lines 212, has been applied to the moving object 202consistent with principles discussed further below. The length of thelines 212 may be proportional to the difference between the magnitude ofthe motion vector 212 of the moving object 202 and the motion vector ofthe POG 208. The direction the lines 212 extend away from the movingobject 202 may be toward the location of the POG 208.

Now consider the case of FIG. 3 , in which the POG 208 follows themoving object 202 as indicated by the viewer motion vector 300. In thiscase, it is assumed that the viewer tracks the moving object precisely,meaning that the difference between the magnitudes of the viewer motionvector 300 and object motion vector 204 is zero. Consequently, minimal(in this case, zero) motion blurring is applied to the moving object202. On the other hand, if desired motion blurring 302 may be applied tothe stationary object 210 because the viewer's POG 208 is moving awayfrom the stationary object 210. The length of the lines 302 may beproportional to the difference between the magnitude of the motionvector of the stationary object 210 (which is zero) and the motionvector 300 of the POG 208. The direction the lines 302 extend away fromthe stationary object 210 may be toward the location of the POG 208.

Refer now to FIG. 4 . Commencing at block 400, head tracking and/or bodytracking (e.g., if the viewer is on a moving platform) may be received,and at block 402 eye tracking may be received. Thus, if desired themotion vector of the viewer's point of gaze (POG) may be determined atblock 404 to be the sum of the motion vectors attributable to bodymotion, head motion, and eye motion, although in some embodiments (e.g.,when the display is a HMD and the viewer's eyes are tracked by aninternal camera on the HMD) only eye tracking need be used to determinethe viewer's POG and any motion of it relative to the display. Combiningthe various motion vectors into a motion vector for the POG may be doneusing vector algebra.

Proceeding to block 406, the motion vector(s) of object(s) on thedisplay being viewed by the viewer is/are determined. This may be doneby accessing object metadata that may accompany rendering of thecomputer simulation.

Block 408 then indicates that based on the difference between the motionvector of an object and the motion vector of the POG, motion blur isapplied (or not). For example, when the motion vectors indicate that theviewer is following a moving object, no motion blur may be applied tothat moving object, whereas if desired motion blurring may be applied toobjects that are not moving on the display on the basis of the viewer'sPOG moving relative to the stationary object. On the other hand, if theviewer is staring at a point in space and not following the movingobject, motion blur may be applied to the moving object.

Motion blurring can be achieved through software by blurring theedges/rendering motion lines as shown in FIGS. 2 and 3 consistent withboth the direction or motion of the POG and the difference between themotion vectors of a display object and the viewer's POG. In some cases,blurring may happen implicitly (for instance, if the viewer is movinghis or her eyes rapidly across a VR HMD, the image may blur in theviewer's eyesight. In some cases, focusing can be implemented byensuring that the portion of the scene that is being tracked by the eyehas reduced or no motion blur.

FIG. 5 illustrates such as VR HMD 500 that may be implemented using anyor all of the components shown in FIG. 1 for the CE device 48, includinga position sensor 502, a motion sensor 504 to sense head motion, acamera 506 to track the eyes, and a transceiver 508 for sending signalsfrom the sensors of the HMD to a motion blur computer 510 such as acomputer simulation console or server.

FIGS. 6 and 7 illustrate a hardware assisted embodiment in which adisplay 600 such as any display herein has the ability to increase thebrightness for only a small portion 602 of the display in which anobject 604 is presented. A brightness control circuit 606 and exposurecontrol circuit 608 may be provided to selectively alter exposure andbrightness for portions of the display 600 such as the portion 602. Forexample, brightness may be increased, and exposure time reduced toreduce motion blur. In the case of VR HMDs, for example, the screen canbe “strobed” to reduce blur, by exposing the image for a short period oftime then having a second period which the display is black. A displaycan have variable exposure time across the whole screen, distributingpower ad heat to only small portions of the screen to achieve higherexposure but for very short periods. As a concrete example, a displaymight show n image at 50% brightness for 20% of the time (and 0%brightness for the remaining time). The same number of photons might bedelivered if that display uses 100% brightness for only 10% of the time.In the moving object (plane) case of FIGS. 2 and 3 , the brightness ofthe plane can be increased, and the exposure time reduced to reduce blur(FIG. 3 ), while doing the opposite for FIG. 2 , i.e., decreasingbrightness for a long exposure, for areas where it is desirable that theeye motion introduce blur.

Blur may be added and removed along one axis and motion due to eyesrelative to the display vs motion due to everything else (head, body,etc.) along the other axis.

For the hardware case, a blur/persistence mask may be added as well asthe frame buffer. This mask informs the hardware how much to persist theimage on a per-pixel/per-region mask. A velocity field as well as theframe buffer may be passed. Asynchronous reprojection may happen closerto the HMD, i.e., the HMD may perform low latency tracking and warp thefield of view right before rendering it, so the viewer gets super lowlatency reprojection, with the blur equivalent of that combining headtracking and/or eye tracking. The HMD could also do a similar actionwith eyetracking where the eye motion is combined with the velocityfield that is passed in to create an up-to-the-moment snapshot of eye topixel motion deciding whether it wants to adjust blur either through asoftware filter running on the HMD or adjusting the exposure of thepixels dynamically.

While the particular embodiments are herein shown and described indetail, it is to be understood that the subject matter which isencompassed by the present invention is limited only by the claims.

What is claimed is:
 1. A system comprising: at least one computer mediumthat is not a transitory signal and that comprises instructionsexecutable by at least one processor to: execute gaze tracking of a userviewing a display; and implement at least one motion blur feature of atleast one object presented on the display according to the gazetracking.
 2. The system of claim 1, comprising the display and the atleast one processor executing the instructions.
 3. The system of claim1, wherein the gaze tracking comprises head tracking.
 4. The system ofclaim 1, wherein the gaze tracking comprises eye tracking.
 5. The systemof claim 3, wherein the gaze tracking comprises eye tracking.
 6. Thesystem of claim 1, wherein the instructions are executable to: identifythe object as moving across the display; identify the gaze tracking asfollowing the object; and responsive to identifying the object as movingacross the display and identifying the gaze tracking as following theobject, implement the motion blur feature as being no motion blur of theobject.
 7. The system of claim 1, wherein the instructions areexecutable to: identify the object as moving across the display;identify the gaze tracking as not following the object; and responsiveto identifying the object as moving across the display and identifyingthe gaze tracking as not following the object, implement the motion blurfeature as being motion blur of the object.
 8. The system of claim 7,wherein the motion blur of the object is proportional to a differencebetween a motion vector of the object and a motion vector of the gazetracking.
 9. The system of claim 1, wherein the instructions areexecutable to: identify the object as not moving across the display;identify the gaze tracking as moving relative to the display; andresponsive to identifying the object as not moving across the displayand identifying the gaze tracking as moving relative to the display,implement the motion blur feature as being motion blur of the object.10. The system of claim 1, wherein the instructions are executable toimplement the motion blur feature at least in part by: establishing afirst brightness and a first exposure in a first region of the displayincluding the object; and establishing a second brightness and/or asecond exposure in a second region of the display not including theobject.
 11. A method comprising: implementing first motion blurring ofan object presented on a display in accordance with a user's gazefollowing the object as the object moves; implementing second motionblurring of the first object in accordance with the user's gaze notfollowing the object as the object moves.
 12. The method of claim 11,wherein the first motion blurring a less than the second motionblurring.
 13. The method of claim 11, wherein the object is a firstobject, and the method comprises: implementing motion blurring of asecond object presented on the display responsive to the user's gazemoving relative to the display, the second object not moving on thedisplay.
 14. An apparatus comprising: at least one display; and at leastone processor configured with instructions to: identify a first motionvector associated with a viewer of the display; identify a second motionvector associated with an object presented on the display; and implementmotion blurring of the object based on the first and second motionvectors.
 15. The apparatus of claim 14, wherein the processor isconfigured with instructions to: identify the object as moving acrossthe display; identify gaze tracking as following the object based atleast in part on the first motion vector; and responsive to identifyingthe object as moving across the display and identifying the gazetracking as following the object, implement minimal motion blurring ofthe object.
 16. The apparatus of claim 14, wherein the processor isconfigured with instructions to: identify the object as moving acrossthe display; identify gaze tracking as not following the object based atleast in part on the first motion vector; and responsive to identifyingthe object as moving across the display and identifying the gazetracking as not following the object, implement greater than minimalmotion blurring of the object.
 17. The apparatus of claim 16, whereinthe motion blur of the object is proportional to a difference betweenthe first and second motion vectors.
 18. The apparatus of claim 14,wherein the processor is configured with instructions to: identify theobject as not moving across the display; identify gaze tracking asmoving relative to the display at least in part based on the firstmotion vector; and responsive to identifying the object as not movingacross the display and identifying the gaze tracking as moving relativeto the display, implement motion blurring of the object.
 19. Theapparatus of claim 14, wherein the processor is configured withinstructions to implement motion blurring at least in part by:establishing a first brightness and a first exposure in a first regionof the display including the object; and establishing a secondbrightness and/or a second exposure in a second region of the displaynot including the object.